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PREFACE 



This report presents interim results of a study conducted under AFHRL Work Unit 
77340207, Development of Methods for Evaluation of Appropirate Grades for Officer 
Positions. This work unit was established in response to a Request for Personnel Research 
(RPR 74-20) submitted by Hq USAF/PRMRE and DPXX, entitled "Development and 
Testing of an Officer Grade Evaluation Technology." In general, the RPR called for 
developing and testing a method by which management engineering teams could 
determine the appropriate grade requirements for individual officer positions based upon 
job content and responsibilities. The technology which was tested represents a 
modification and extension of methods developed during the 1963-1966 time period 
under Project 7734, Development of Methods for Describing, Evaluating, and Structuring 
Air Force Occupations; Task 773402, Development and Appraisal of Methods for Job 
Evaluation. The present investigation draws freely from a series of reports which 
documented the original Officer Grade Requirements (OCR) project. 

Due to the scope of the inital OCR project, it was not feasible to fully describe the 
OCR project in a single report. However, for a general overview, the first of the series of 
reports (Christal, 1965) is recommended. This first report also mentions tlie many 
scientific, military, and support personnel who contributed to the OGR study, and the 
author wislies to again express his appreciation to these individuals. 

The author served as general director of the present study and was responsible for 
writing this report. However, the actual work to accomplish the requirements of RPR 
74-20 must be credited to the following individuals: 

Hq USAF - Maj Joseph S. Kittle, RPR Requirements Manager, AF/PRMRE 

AFHRL - Dr. Joe T. Hazel, Chief, Career Development and Job Evaluation 
Branch, Project Monitor 

Mr. William J. Stacy, RPR 74-20 Principal Investigator 
Dr. Gary N. Matthews, Task Scientist 

Col James B. Carpenter, Chief, Occupational and Manpower Research 
Division 

Mrs. M. Joyce Giorgia, Scientific Assistant 

Mrs. Maria M. Courtney, Cold Type Composing Machine Operator 

Sgt Johnny J. Weissmuller, Lead Computer Specialist 

AlC Bruce B. Barton, Computer Specialist 

This report essentially describes the continuity between the initial OGR project and 
research results accomplished under RPR 74-20. Other reports will follow which give 
more detailed information concerning application of the method by management 
engineering teams. This study does not constitute authority to alter existing Air Force 
officer grades. Further action towards such objectives is a function of Hq USAF, and is 
recommended only after certain prescribed additional research. 
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SYSTEMATIC METHOD FOR ESTABLISHING OFFICER 
GRADE REQUIREMENTS BASED UPON JOB DEMANDS 



I. INTRODUCTION 

This report describes the developnient and testing of a technology by wliich management engineering 
teams (MET) can deteniiine the appropriate grade requirements for individual officer positions based upon 
job requirement factors. It represents a modification and extension of research previously conducted under 
AFSC Project 7734. Tliis earlier research will be described in order to facilitate the reader s understanding 
of the present study. 



II. BACKGROUND 

There was a time when determination of grade requirements was a simple matter. In the Roman 
army a consul commanded two legions. Each legion had a standard complement of officers, including six 
tribunes and sixty centurions. Even in the American anny, grade determination was not a problem until 
sometime after the Civil War. Tlie fighting force was composed primarily of cavalry and foot soldiers, and 
grade was a simple function of the size of the unit commanded. But advances in technology have led to the 
creation of many new jobs wliich cannot be graded simply on the basis of size of the unit commanded. 
Ground rules for attaching a grade level to a particular job ^ have become less and less clear. 

Rather than justifying grade requirements on the basis of job requirements, in recent years there has 
been a tendency for the Armed Forces to request and defend grade authorizations to support career 
planning objectives. The philosophy is that qualified officers normally should be promoted after specified 
intervals of time, and specific numbers of authorizations must be maintained at each grade level in order to 
avoid promotion stagnation and to insure orderly career progression. Congress and tlie Office of the 
Secretary of Defense (OSD) have concerned themselves with equating promotion opportunities in the 
various services, and have looked at grade as the vehicle for awarding and controlling pay. The result is that 
the relationship between job requirements and grade requirements became rather vague. 

A study was conducted by the Personnel Research Laboratory^ during the 1963- 19$5 time period, 
in wliich an attempt was made to determine the appropriate distribution of officer grades for the Air Force, 
based upon job requirements. Tliis was one of the largest, most systematically accomplished, and best 
documented job evaluation studies ever accomplished. It involved application of a "policy capturing 
model"^ (Christal 1963, 1968) and yielded a set of job evaluation factors which simultaneously defined 
the meaning of grade and provided a basis for evaluating officer positions in terms of their grade 
requirements. The Officer Grade Requirements (OGR) Project provided the basis for constructing the grade 
"yardstick" currently proposed for application by management engineering teams. For this reason it will be 
discussed in some detafl. 



I!!. niH OFFICER GRADE REQUIREMENTS (OGR) PROJECT 

The Officer Grade Requirements Project was conducted in tliree phases, as follows. (1) obtaining 
policy decisions concerning the appropriate grades for a selected "criterion" sample of jobs (Hazel, 1965), 
(2) development of an OGR policy equation which assigns the same grades to these jobs as was given by the 
PoHcy Board (Hazel, Christal, & Hoggatt, 1966), and (3) application of policy equation to jobs in the Air 
Force population to determine the total distribution of officer grade requirements. 



In this report die words *'job*' and "position" arc used interchangeably. 
^Later integrated into tfie Air Force Human Resources Laboratory (AFHRL). 

^Subsequently used by scientists in many important applications. A form of tliis model was used to develop the Atr 
Force Weighted Airman Promotion System (WAPS). 
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Phase L Policy Board Grade Ratings for Jobs in the Criterion Sample 



Development of an objective system for grade determination depends upon a stable statement of 
appropriate grades for a sample of jobs. The grade levels assigned to jobs in tliis criterion sample by a USAF 
Policy Board are used to calibrate the job evaluation system, which in turn is applied to the remaining bulk 
of Air Force jobs. They are like the benchmarks maintained by tlie Bureau of Standards to calibrate 
surveyor tapes; they must be accurate and unbiased, since they become the Air Force standards for 
determining grade requirements. 

Six steps were involved in the first phase of the OGR project: (1) collection of detailed information 
describing approximately 80,000 officer jobs; (2) selection of a "criterion" sample of three to four 
thousand descriptions representing all levels and types of Air Force officer jobs; (3) selection of a USAF 
Policy Board, (4) obtaining grade ratings for jobs in the criterion sample from members of the Policy Board, 

(5) analysis of the Policy Board's ratings to detennine that they are reliable, reasonable, and unbiased, and 

(6) acceptance or rejection of tlie ratings as a basis for establisliing Air Force policy concerning grade 
determination. Each of these steps is discussed. 

Collection of Job Descriptions 

Tlie first major step of the project was to collect accurate and detailed information describing the 
work performed by Air Force officers in grades lieutenant through colonel. Preliminary forms and 
instructions were drafted and tried out at Langley Air Force Base in late August 1963, and final forms were 
reproduced and sent to the commands in November 1963. Job descriptions were obtained from all officer- 
except generals, student officers, patients, air attaches, officers in the medical-professional and dental 
utilization fields, and selected officers in the operations career area, where the number of incumbents is 
large and where many jobs are known to be identical. In all, descriptions were received from 79,750 
officers. 

Each incumbent was asked to provide a job title, a verbal description of the location of his job in the 
Air Force organizational structure, a detaOed description of his job in terms of the duties and tasks he 
performs, and information describing any unusual requirements of the job not reflected m his duty and task 
statements. In addition, he was required to provide the name and base location of his organization, the 
UMD'* authorized grade for his position, his present grade and the grade of his supervisor, his primary and 
duty AFSC; the length of experience in his present job, in his duty AFSC, and in the Air Force; the 
organizational level of his job and the level of liis job within his organization, his command affihation,liis 
duty phone numbers, and his educational background. The incumbent's supervisor was asked to review the 
completed job description and supplementary information for accuracy and completeness and to provide 
his judgment concerning tlie appropriate grade authorization for the job. 

Selection of the Criterion Sample 

Upon receipt, the job descriptions were numbered, sorted into UMD authorized grade-by-AFSC 
categories, and counted. A representative criterion sample consisting of 3,575 cases was selected in tlie 
following manner, (a) two descriptions were pulled at random from each UMD grade-by-AFSC category for 
which SIX or more officers were assigned, (b) one description was pulled from each category containing less 
than six but more than two officers assigned, and (c) additional cases were selected to supplement those 
categories where the number of incumbents was large or where the AFSC was known to contain a large 
number of job types. 

Selection of the Policy Board 

A Policy Board which included representatives from twelve major commands was selected by Hq 
USAF. The Board was composed of 22 colonels, and Lt Gen James E. Briggs, USAF (Ret), was appointed 
chairman. Each member was selected on the basis of his overseas and zone of interior experience in 
particular career areas. For any one of the 3,575 jobs in tlie criterion sample, there was at least one board 
member who could serve as an expert consultant to tlie rest of the board. 

'^Currently referred to as UDL authorized grade. 
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Ratings by the Policy IJoard 

Tlie Policy Board was convened at \k\ USAF on 10 February 1964 for five days. During this period, 
members of the board were charged with the responsibility for determining the appropriate grade level for 
each of the 3,575 jobs in the criterion sample. Each job was rated independently by five members of the 
Policy Board. 

The following measures were taken to assure that ratings would be reliable, valid, and unbiased. 

a. 77/e board members were instructed by General Briggs and the Director of M^O concerning the 
importance of their mission. 

"This project will provide data to support decisions concerning an optimal grade structure for the Air 
Force. The judgments made by this board during the coming week will play an important role for a long 
time in the future in determining the number of colonels, lieutenant colonels, majors, captains, and 
lieutenants needed by the Air Force to carry out its mission. 

. . it is essential for you to be as objective and impartial as possible. This board has been convened 
to make an accurate statement of Air Force grade requirements. It is important that members resist any 
tendency to exaggerate these requirements."^ 

b. The board had access to any information needed to make accurate judgments. 

Board members were asked first to rate the appropriate grade level for a job and then to indicate on a 
3-point scale their level of confidence in such ratings. 

If they needed more information concerning a particular job, the following sources for additional 
information were available. 

(1) Board members could consult the one or more members on the board with experience in tlie 
career area containing the job. 

(2) Organization level, command, and base or installation of a job was furnished any member on 
request. Members could obtain organizational charts from clerks in the room for commands and 
subcommand units in order to better identify the organizational location of the job. 

(3) Members could telephone special air staff consultants in Hq USAF. 

(4) Members could telephone the supervisor of the incumbent in the job being rated. 

Ratings were to be independent and were to reflect the unbiased judgment of the rater alone. Tlie 
board members were not allowed to have knowledge of the current UMD grade authorized for the job being 
rated nor of the grade stated by the incumbent's supervisor as being appropriate for the job. They were not 
informed of the grade held by the incumbent in the job or by his supervisor. Board members were 
instructed not to ask any other board member or consultant concerning the appropriate grade for the job. 

c. Grade ratings for particular jobs were obtained independently from five separate board members. 

Previous research indicated that the average of five independent ratings would provide a stable 
estimate of grade requirements (Christal, Madden, & Harding, 1960). 

d . Each job was rated in a context of other fobs. 

Research on context effects (Madden, 1960) has indicated that most accurate ratings of job level are 
obtained when a job is considered with other jobs of varying content and level. When a job is rated with 
high level jobs it tends to be underestimated; when it is rated with low level jobs it tends to be 
overestimated, when it is rated alone, the obtained ratings tend to be unstable. Care was taken in the OGR 
study to vary the types and levels of jobs in each rating booklet to control the effects of context. 

e. Board members were required to rate grade requirements using a 16-point rating scale. 

Rating results are more stable when judges make the finest discriminations of which they are capable. 
Most officers can easily distinguish between a job which is appropriately filled by a junior colonel and one 
which should be filled by a colonel with considerable experience and time in grade. Therefore, in this 
project, Policy Board members were required to use a l6-point rating scale (Table 1). This scale recognizes 
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three IcvcL uf CApcncncc rci[uircnicnts within each grade from lieutenant thruugli colonel, and one level for 
general. 

Table L Criterion Board Rating Scale 



Grade Code 



General 




16 


Colonel 


Senior 


15 


Colonel 


Middle 


14 


Colonel 


Junior 




Lt Colonol 


OvlllUl 


1 9 
1 ^ 


Lt Colonol 




1 1 
1 1 


Lt Colonel 


Junior 


10 


Major 


Senior 


9 


Major 


Middle 


8 


Major 


Junior 


7 


Captain 


Senior 


6 


Captain 


Middle 


5 


Captain 


Junior 


4 


Lieutenant 


Senior 


3 


Lieutenant 


Middle 


2 


Lieutenant 


Junior 


1 



Analyses of the Policy Board Rating Data 

Since ratings provided by the Policy Board were to be the basis for establishing Air Force grade 
requirements, it was important to den[K)nstrate that tliesc ratings were stable, tliat there was high agreement 
among board members concerning the appropriate grade requirements for particular jobs, that the raters 
had confidence in their ratings, and that the raters were not biased for or against jobs in various AFSCs or 
commands. To this end, a scries of analyses of the Policy Board ratings was accomplislied. The main results 
of these analyses are summarized in the paragraphs below: 

a. Tliere was high agreement among the Policy Board members concerning grade requirements for 
jobs in the criterion sample. The reliability coefficient for the mean grade ratings provided by the Policy 
Board was .92 (Linquist, 1953, p. 361), a figure which indicates a liigli level of agreement. Another statistic 
useful in mterpretation of the results is the standard error of estimate, wliich was .79 for the criterion board 
data (on the 16 point scale). Tliis value indicates that, if judgments of a large number of similar' boards were 
obtained, 95 percent of the mean grade ratings would be within plus or minus onc'half of a grade level of 
the mean grade ratings computed from the policy Board data (± 1.55 on the 16-point scale). 

b. Members of the Policy Board expressed confidence in their ratings of job grade requirements. 

Figure 1 presents a distribution of mean level of confidence expressed by board members in their 
ratings of 3,575 jobs in the criterion sample. For 2,387 of the jobs, at least four of the five raters expressed 
tlie liigliest level of confidence in their judgments. Only 59 of the 3,575 jobs had an associated mean 
confidence rating of less than 2.00 on the 3-point scale. Board members tended to have most confidence in 
ratings of well established jobs, such as those in the aircrew areas. They showed somewhat less confidence 
in ratings of the newer and less well established jobs, such as those in the scientific areas. 

c. Individual members of the Policy Board did not exhibit a bias toward fobs in particular commands 
orAFSQ. 

Table 2 shows summary results of an analysis designed to identify raters exliibiting a bias for or 
against jobs in a particular command or occupational grouping. The values in tliis table were computed by 
taking the difference between the average of ratings assigned by a rater (on tlic 16-point scale) to jobs in a 
particular category from an average of ratings assigned by all raters to jobs in that category. Since tliree 
points on the 16-point scale represents one grade level, a value of 3.0 in Table 1 would indicate that a board 
member rated jobs in a given category approximately one grade higlier than other board members. 
Similarly, a value of -3.0 would indicate judgments averaging approximately one grade lower than those of 
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Fig. 1, Confidence of board members in grade judgments. 



ERIC 



12 



X 

8. 

O 

U 



C 
O 



E 
2 



o 
o 
fiQ 

u 
o 
Ui 



> 
< 



o 
U 



oodddodddddddddd 

• • III I I I I I I I I 

d d d d d d d d d d d d 

• I I i I I I I I I I I I I I I 

I I I I 

<s<NvO<N»r»0'-<ooo^»r»or^c^ooo 

dddddddddddddd»^d 
I I I I III 

oooooooooddodoo'o 

><r»-< <N -H CS (A 00 XT -H -H r<^ C\ O rr» 

ooooo'-^oooodod'^dd 
I 

rc»rc»xrfNfN{Nrt~».-HXrxrr^OfNO»^0 

oooooooooodododd 

oooooooooo oodddd 
iiii iiiiiiiiiii 

xroor* rc»^orr»>«rr-rr»r-fNxrfNxroo 
oooooooodddd^ddd 

oooooooooooooood 

w^OOOOO^OO'-<00-^r^\OOOOOOVOw^ 

Oi-«ooooooooooddo d 
till I I I I I I III 

O C\ r<^ O r>» — ' C\ Cn O O rCiOOO w^w^r» 

vor>.'<rr^i-ifNi-i wn>»^><r ovOvo— <oo rr» 

O'-'OOOOOOOO'^'^OOO o 
I I I I I 

00000i-<00000000'-<0 
I II 

oOr-iooo"<rr^r^CNO<NCNrsjoofNrr» 
p^r^r^O'-^'-^ddd'-^'-^d'-^O'-^d 

lltlllllltlltlll 

fNfN<NfNrriOO\0-<\0'^ rri'<r'<rxr\0'-< 

dddodooddddd^odd 





rsi 


VO 








O 


\0 


\0 rci rci 








o 


o 


o 


o 


O 


o o 


o 


o o 


O O o 


O 




o 


CM 








00 










Cs 


o 





OfNr-<fNrC»00rC»><r^fNfNOrr»N<r'-^rs> 

o d d d d r> d d d d d d d d d d 

C\ ^ O 00 00 u^rslCOVOC\f^vO00VOfNI'-< 

©•--••-•ooooooo •-iood»-<o 

I 

^r»fNfN^fNrc»i-H><r^'<roo^r-ii-H>«r 
ooddoddoododdd^d 



\ 



CO 

(/) 
< 

(I, D 

< cr 

3 = ^ 

< P. U 



S3 



'^u o u o 
=; < < < o-iu 

< to H CU < 



< 

uj 8 

• to 
U • 
•J u 
lu < 
< < 



< 

< U 

< O 



op 
'> 

2 

T3 



00 O4 



S '< to 



o 
to 

S 

c 

I 

< 



X 



X 



o 

& 

u 

H 
1» 

X 

X S 

tr» i-i 
C 

c c« 

X 00 



sT X 
X X 

X S 



X 
X 



X VO 



X 
X 

X 
X 



X 

^ U 
S u. 

2 ■= 
§ S. 



u 

to 

< . 

•c X 

c X 



X 
X 



X 
X 



X 
X 



X 
X 

C\ 

X* 
. X 

X JO 

X ^ 

g x"^ 



X X S 

5 E 
^ 

X X 

X X X 

^ C\ ^ 

1^ X X 

s ^ 

S ^ ° 

X X 

X s ?s 

X 

^ X* « 

« X o 

c\ to 

U 00 (I4 
to , 
X 



X 
X 



c 



X 
X 



< X 



tS X* 

T3 X 
c *^ 



00 5« 

M to 



to 

< 

C It 

.2 3 



10 



ERIC 



13 



other members. Tlie highest reported value is only 1.7, and most of the values are less than 1.0. Tlie largest 
values tend to be associated v^ith judges who rated aU categories somewhat liigh or low, and these judges did 
not show a bias toward jobs in particular categories. 

d. Policy Board members agreed that many officer jobs are inappropriately graded. 

Detailed analyses of the Policy Boards ratings indicated that each job was considered on its own 
merits. There was no systematic tendency on the part of raters to confirm current UMD grade 
authorizations or to inflate their ratings of grade requirements. Many jobs were downgraded as much as one 
or two full grade levels. Others were upgraded. Futhermore, the reliability analysis indicates there was 
strong agreement among board members as to which particular jobs should be upgraded or downgraded. 

Acceptance of the Policy Board Ratings 

The Persbnnel Research Laboratory was prepared to abort the OCR study, if for any reason the 
Policy Boai-d's ratings were not defensible. Fortunately, the ratings appeared to be defensible from every 
point of view. To summarize: 

a. The board was composed of experienced colonels who had a clear concept of the meaning of 
military grade as related to Air Force jobs. 

b. Board members knew the jobs they rated. They had access to job descriptions, organization 
information, teclinical consultants, and job incumbents or supervisors, as necessary. 

c. Board members were confident in their grade ratings. 

d. Board members were not biased toward jobs in particular commands or specialties. 

e. Board members agreed with each other concerning the appropriate grade level of particular jobs. 

f. Board members did not give inflated ratings and did not simply confirm current UMD 
authorizations. Each job was rated on its own merits. 

Since the board ratings were acceptable as standards for grade determination, the Personnel Research 
Laboratory proceeded with the second phase of the OGR Project - the development of a mathematical 
equation to express the board policy in terms of weiglited measures of job characteristics. To be acceptable, 
this equation must assign the same grade to a job as did the Policy Board. Its effectiveness can be gauged by 
the extent to which it predicts the grades assigned by the board to the 3,575 jobs in the criterion sample. 

Phase IL Dcvciopment of the Officer Grade Requirements Policy Equation 

Success of the OGR study hinged on development of a mathematical equation which would predict 
grade ratings made by the Policy Board. Four steps were involved in this phase of the study: (1) 
hypothesizing the job requirement factors considered by Policy Board members in making their ratings, (2) 
evaluation of the criterion jobs in terms of these factors, (3) development of a policy equation which 
weights the job requirements factors into a composite to predict the Policy Board's grade ratings, and (4) 
evaluation of this policy equation. Each of these steps is discussed next. 

Hypothesizing Job Requirement Factors Considered by the Policy Board 

Perhaps the most challenging part of the OGR study was identifying the factors considered by the 
Policy Board in making their grade decisions. The fact that board members agreed with each other, 
concerning the appropriate grade requirements for jobs in the criterion sample, indicated they considered 
the same or highly related factors in making their decisions. 

It was hypothesized that four classes of variables might be used to predict the Policy Board ratings: 

a. Job requirement factors. These are demands of the job which have a bearing on grade 
determination. Interviews with members of the PoHcy Board provided a number of clues concerning factors 
which influenced their judgments. Over and over again, board members mentioned factors such as the 
complexity, variety, and level of activities which must be managed by the job incumbent; the possible 
impact of decisions made by the incumbent on Air Force mission, the types of planning activity required of 
the incumbent; and the types of knowledges and experiences which the incumbent should possess. 
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b. Factors relating to the position of the job in the Air Force organizational structure. It was 
hypothesized that the location of the job in the Air Force organizational structure would have a bearing on 
grade requirements. 

c. Coincidental predictors. It was hypothesized that certain non-relevant factors, such as the number 
of words in the job desaiption or the verbal facility of the description writer, might have influenced judges 
in their rating of a job. 

d. Command or specialty affiliation 

Evaluation of the Criterion Jobs in Terms of Hypothesized Predictors 

Data concerning some of the factors to be considered for inclusion in the policy equation could be 
obtained from the job description forms. These variables included such things as the supervisor's grade 
rating, the oiganizational level in which the job occurs, and the level of the job within this organization. 
Previous research (Christal & Madden, 1961; Hazel & Madden, 1965; Madden, 1963a; Madden, 1963b) and 
interviews with members opthe Policy Board indicated that certain job evaluation factors, such as the types 
of decisions and judgments required by the job would have relevance for grade determination. These factors 
could only be measured through use of rating scales, and a procedure had to be established for obtaining 
such ratings. The 3,575 job descriptions were sorted into 143 booklets, each containing 25 job descriptions. 
Each booklet was sent to five or more majors or lieutenant colonels in the field, who were selected at 
random throughout the Air Force on the basis of the last two digits of their service number. Over 700 
officers participated in this phase of the study. Each officer rated each job on the ten job evaluation factors 
listed in Appendix A. He also rated each job on the same 16*point scale which was used by the Policy Board 
in making their grade determination. 

A mean rating score for each of the ten job evaluation factors and the grade rating were computed for 
each job. These scores were utilized, along with other variables, in deriving the final policy equation. 

Derivation of the Policy Equation for Grade Dctcrraination 

Success of the OGR study hinged on the development of an equation which accurately predicts grade 
ratings made by the Policy Board. In developing this equation, nearly two hundred predictor variables were 
considered (Appendix B). The final equation contains nine predictors which accurately reproduce the board 
policy. These nine predictors include five of the job evaluation factors listed in Appendix A, two predictors 
associated with organizational level of the job, the mean grade rating obtained from field judges, and the 
supervisors' recommended grade level. They are defined as follows: 

a. Management: The level of executive, and managerial skills required by the job. Consider the 
complexity, variety, and level of the activities which are directed, organized, coordinated, controlled, 
commanded, or evaluated. (Ratings for each job were obtained from five field judges. Correlation between 
mean ratings and Policy Board ratings = .75.) 

b. Planning: The extent to which planning is required by the job. Consider the scope and 
significance of work for which planning is done. The longer the time span for which planning is done, the 
higher the rating should be. (Ratings for each job were obtained from five field judges. Correlation between 
mean ratings and Policy Borad ratings = .71.) 

c. Special Training Work Experience: The extent to which the job requires knowledges and skills 
which must be acquired through special training courses or on-the-job experience. Disregard general courses 
given by Squadron Officer School, Command and StaffCollegc, or War College. (Ratings for each job were 
obtained from five field judges. Correlation between mean ratings and Policy Board ratings = .45.) 

d. Judgment and Decision Making: The importance and independence of judgements and decisions 
required by the job. Consider the nature, variety, and possible impact of decisions. The less well-defined the 
guidance for decisions, the higher should be the rating; while the more specific and detailed the guidance, 
the lower should be the rating. (Ratings for each job were obtained from five field judges. Correlation 
between mean ratings and Policy ratings = .60.) 

e. Communication Skills: The extent to which the job requires skill in oral and written 
communication. Consider the complexity and variety of information communicated, as well as the level of 
the individuals and agencies involved. (Ratings for each job were obtained from five field judges. 
Correlation between mean rating and Policy Board ratings = .63.)^ 
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f. Level of Organization in Which Job Occurs: The level of the organization in which the job occurs, 
defined by the following scale. (Correlation with Policy Board ratings = .53.) 



g- Level of Job Within, Organization. The level of the job within the organization in which it appears, 
defined by the following scale. (Correlation with Policy Board ratings = .53.) 



h. Mean Grade Rating by Field Judges: The mean of grade ratings for each job obtained from five 
field judges in the field. (Correlation with Policy Board ratings = .89.) 

i. Supervisor's Judgment of Appropriate Grade: A statement from the supervisor of the job 
concerning its appropriate grade level. This correlates .63 with the Policy Board ratings when it is used as a 
single scaled variable. Since it is more predictive when broken down into six categorically-coded variables, it 
was used in the equation as follows: 

(1) 1 if rated general by supervisor; 0 otherwise 

(2) 1 if rated colonel by supervisor; 0 otherwise 

(3) 1 if rated // colonel by supervisor; 0 otherwise 

(4) 1 if rated major by supervisor; 0 otherwise 

(5) 1 if rated captain by supervisor; 0 otherwise 

(6) 1 if rated lieutenant by supervisor; 0 otherwise 

« The Policy Board ratings can be predicted very well using the five job evaluation factors in 
combination with the two organization variables (correlation with Policy Board ratings = .84). It would be 
difficult to prove that these particular variables were the ones considered by the Policy Board in making 
their grade judgments. Since these seven variables have high "face validity" for grade, and they accurately 
predict the Policy Board's decisions, it can be assumed that they are primary determiners of grade 
requirements. 

While the mean grade rating from five field judges had the highest relation with tha criterion for a 
single variable, one should recognize that in making their ratings these judges also considered information 
measured by the job evaluation factors, such as the management and organizational levels of jobs, along 
with their requirements for planning, decision making, special work experience, and communication skills. 
In fact, the five job evaluation factors in combination with the two organizational variables correlated .92 
with the grade ratings obtained from all field judges. Even so, grade ratings obtained from field judges make 
a unique contribution in predicting the Policy Board ratings. This is believed to be due to a large number of 
special factors associated with specific jobs which were considered by both the Policy Board and the field 
judges in arriving at grade requirements. Since any one of these factors may apply to only one or two jobs, 
it would be uneconomical to develop scales for applying them to all jobs. 

The supervisor's judgments concerning grade makes a very small unique contribution to the system, 
and this contribution is believed due to an occasional job requirement factor considered by the supervisor 
and the Policy Board, but not considered by the field judges. 

It is of interest to examine some of the variables which add nothing to the nine selected predictors in 
the final policy equation. These include the foUovwng: 



DODorHq USAF 
Hq Major Air Command 
Numbered AF'or equivalent 
Air Division or equivalent 
Wing or equivalent 
Group or equivalent 
Squadron or equivalent 
Detachment or equivalent 
Other 



9 
8 
7 
6 
5 
4 
3 
2 



Command Element 

Directorate, I>epartment, Office or equivalent 

Division or equivalent 

Branch or equivalent 

Section or equivalent 

Unit or equivalent 

Other 



7 
6 
5 
4 
3 
2 
1 
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a. 


Variables associated with AFSC affiliation 


b. 


Variables associated with command affiliation 


c. 


UMD authorized grade 


d. 


Grade of the incumbent 


e. 


Grade of the incumbent's supervisor 


f. 


Variables expressing interaction between AFSC and organizational level 


g- 


Educational level of the incumbent 


h. 


Length of the job description 



The fact that variables associated with AFSC and command affiliation did not enter into the policy 
equation is additional evidence that the Policy Board was not biased toward jobs in any one of these 
categories. While some of the variables listed previously (such as UMD authorized grade) are related to the 
criterion, their valid variance is already covered by one or more of the nine predictors included in the 
accepted policy equation. 

Evaluation of the Policy Equation 

When the policy equation was applied to jobs in the 3,575 case criterion sample, it was demonstrated 
to be highly accurate. In fact, data in Table 3 show that the policy equation pinpointed 82 percent of the 
jobs in this sample within one-third of a grade level of the ratings assigned by the board on the 16-point 
scale and that the policy equation predicted within two-thirds of a grade level for 96.7 percent of the jobs. 
The correlation coefficient between grades allocated by policy equation and grade ratings provided by the 
Policy Board was .92. 



Table J. Accuracy in Prediction of USAF 
Policy Board Job Ratings 



Degree of Accuracy 


Cumulative N 


Cumulative % 


Exactly on 


1,292 


• 36.1 


Within 1/3 of a grade 


2,931 


82.0 


Within 2/3 of a grade 


3,457 


96.7 


Within 1 grade 


3,557 


99.5 


Within 1 1/3 of a grade 


3,572 


99.9 


Within 1 2/3 of a grade 


3,575 


100.0 



Since a given job was rated by only five members of the Policy Board, it is probable that in a few 
instances individual inconsistencies and policy peculiarities combined to cause a job to receive a grade rating 
different from that which would have been obtained had the entire board rated the job. The policy 
equation represents the best single statement of policy for the board as a whole, and it was applied in a 
precise and invariant manner to every job. An "error" made by the policy equation in predicting the Policy 
Board's rating of a job does not necessarily indicate that the equation assigned a grade different from the 
one which would have been obtained had the entire board rated the job. In any event, such "errors" were 
few in number and small in magnitude. Furthermore, these errors were randomly distrubuted within AFSC 
categories. That is, the distribution of grade levels assigned to jobs in an AFSC category by the policy 
equation was in all instances approximately the same as the distribution of grades assigned to jobs in that 
AFSC category by the Pblicy Board. 

As stated before, the nine variables in the final equation were selected after examining more than a 
hundred potential predictors. Some readers may wonder if this process did not involve a capitalization on 
chance relationships. To evaluate this possibility, the criterion observations were randomly divided into 
subsamples, A and B. Least squares weights for the nine final predictors were developed in a sample A and 
cross-applied to sample B. Similarly, weights were developed in sample B and cross applied to sample A. As 
indicated in Table 4, the resulting shrinkages in multiple correlations (Rs) averaged less than one point in 
the third decimal place. 
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Table 4, Cross Application of the OCR 
Pblicy Equation 



O«vtlopmtnt 
Sampit 



N 



Application 
Sampit 



Multiplf 
R 



A 
B 

A + B 



1,788- 

1,787 

3,575 



B 
A 
A + B 



.917 
.918 
.917 



In summary, all data indicate that the policy equation is stable and that it does an excellent job of 
assigning grades judged appropriate by the Policy Board. Errors were few in number, small in magnitude, 
and randomly distribured. In view of these findings, the OCR Project moved to the next phase, application 
of the policy equation to determine the appropriate grade levels for an additional 10,000 officer jobs. 



Application of Equation 

After the policy equation had been developed; and demonstrated to assign the same grades to jobs as 
did the Policy Board, Phase III was undertaken. This phase involved application of the equation to 
determine the appropriate grade levels for a sufficiently large number of jobs to serve as a base for 
estimating the appropriate distribution of grades in various specialties and specialty groupings. Ten 
thousand jobs were selected for inclusion in this base sample. Of thesc^ 1,750 were selected from the 
original 3,575 criterion job sample so the policy equation could be revalidated. The remaining 8,250 jobs 
were new jobs selected from the 79,750 population file to provide adequate representation of every 
AFSC-by-UMD grade category. Descriptions for the 10,000 jobs were randomly sorted into 400 booklets, 
each containing 25 job descriptions. Each booklet was rated by not fewer than five majors and lieutenant 
colonels, who were selected at random throughout the Air Force on the basis of the last two digits of their 
service numbers. Over 2,000 officers participated in this phase of the project. Each officer rated each job in 
his booklet on the appropriate job evaluation factors, and the same 16-point scale as used by the Policy 
Board in making its grade determination. 

Mean score values on the job evaluation factors and field grade ratings were combined by the, policy 
equation with organizational variables and supervisory grade ratings to determine the appropriate grade 
level for each of the 10,000 jobs. The grades assigned by the policy equation to the 1,750 criterion jobs 
were compared with the grades assigned to these same jobs by the policy equation using ratings collected 
from field judges in the development sample. These two sets of predicted grades correlated .93 and had 
approximately equal means and standard deviations. Thus, the policy equation was shown to have high 
stability across time and judges. 

Projection of Results 

At this stage of the OGR Project, appropriate grade requirements had been determined for 11,825 
officer jobs. Of these, direct ratings from the Policy Board were available for 3^75 jobs, and ratings 
resulting from application of the policy equation were available for 8,250 jobs. This 1 1,825 was used as a 
base to determine the appropriate distribution of grades for various specialties and specialty groupings. The 
method for accomplishing these projections is described by Christal (1965). 

Results of the OGR Project 

Results of the OGR Project projections indicated that changes in grade allocations would have to be 
made in many officer utilization fields to bring statements of grade requirements into line with job 
demands. In some utilization fields grade requirements appeared to be grossly overstated, in others they 
were understated. In every utilization field some jobs were over classified while others were under classified 
by grade level. 



Phase III. Application of the Policy Equation to the 10,000 
Case Sample and ftojcction of Results 
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For example, the study indicated that too many colonel positions were allocated to the Missile 
Operations, Safety, Weather, Missile Maintenance, Manpower Managenwnt, Education and Training, 
Information, and Legal utilization fields. At the same iimCy too few colonel positions were allocated to the 
Operations, Aircraft Control, Weapons Director, Scientific, Research and Development, Development 
Engineering, Communication-Electronics, Avionics/Munitions, Aircraft Maintenance, Civil Engineering, 
Trasportation, Logistics, Procurement, Financial, Administrative, Intelligence and Air Police utilization 
fields. 

The OCR Project results indicated that the Air Force could get by with considerably fewer lieutenant 
colonels in Pilot and Navigator cockpit positions. It sliould be remembered however, that lliis project did 
not attempt to determine the appropriate pay levels for these positions, which would have demanded 
consideration of factors such as working conditions and risk. Nor did it consider the need for the Air Force 
to keep rated officers in the cockpit in order to amortize the cost of their training, which is considerable. 

Overall, the OCR Project indicated that the Air Force was somewhat undergraded at the colonel and 
lieutenant colonel levels in 1964, and considerably undergraded at the major level. Readers who want more 
detailed information about study results should refer to the references. 

Application of OGR Results 

The Officer Grade Requirements Project results had an impact on the allocation of grades to officer 
positions, but they have never been implemented across-the-board. A number of lieutenant colonel cockpit 
positions were downgraded to major. Major positions in other areas, such as communication-electronics, 
were upgraded to lieutenant colonel. OGR results were used to evaluate requests for upgrading from the 
field for several years after the study was completed. Such requests were routinely rejected if they were not 
supported by OGR Project findings. The OGR Project also was instrumental in convincing the Department 
of Defense of the need for supporting and Air Force request for temporary relief at the major level. 



IV. DEVELOPMENT OF BENCHMARK SCALES 

The 1964 OGR Project produced a statement of the appropriate distribution of grades for each 
officer utilization field and for the Air Force as a whole. However, it did not yield a grade evaluation 
procedure which could be applied to individual officer positions. Previous research on context effects 
(Madden, 1960) indicated that the most accurate ratings of job level are obtained when a job is considered 
with other jobs of varying content and level. As stated earlier, when a job is rated with high level jobs it 
tends to be underestimated; when it is rated with low level jobs it tends to be overestimated; when it is 
rated alone, the obtained ratings tend to be unstable. In the OGR study, context effects were controlled by 
making sure that when job or job factor ratings were collected, the rater always executed his judgments on 
a carefuUy selected set of jobs. 

In order to develop a system for evaluating individual officer positions, it first was necessary to 
construct job-factor scales with carefully anchored rating levels. That is, the rating scales had to provide the 
rater with an appropriate context within which he could nest his judgments. In the case of the OGR factors, 
a decision was made to develop scales using generally recognized job titles as benchmarks for level 
definitions. The efficiency of the scales could then be tested through application to a sample of jobs from 
the original OGR study. This effort is documented in an available publication (Brokaw & Giorgia, 1966) 
and will be dealt with only briefly here. 

Selection of Job Titles 

The list of 1 1,789 jobs evaluated in the Officer Grade Requirements Project were ordered in terms of 
their mean ratings on each of the ten job evaluation factors, Then each ordering was divided into nine 
sections, from high to low. Finally, eleven job titles were selected from each of tlie nine levels on each 
factor. These titles were selected to have minimum standard deviations and were so selected as to emphasize 
the range of ratings given. That is, for the fifth, or median group, the eleven titles were selected as near 
center of the range of means involved as possible; for groups other than the fifth, titles were chosen from 
tlie portion of the range remote from the median. This had the effect of identifying the highest possible 
mean rating for titles for use as the top or "9" category, and the lowest possible mean rating for titles for 
use as the lowest or "T* category. 
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Decks of inanila cards were prepared for each factor. A deck contained 99 cards, each of which listed 
Uie title for one officer position, along with its organizational level and location. These decks were sent to 
volunteer rating officers (majors and lieuteuance colonels) who were asked to order the jobs in a deck in 
terms of their requirement levels for a particular factor. 

Intraclass correlations of the ratings on each factor were obtained for the 99 jobs (Haggard, 1958). 
The numbers of raters and factor reliability coefficients are reported in Table 5. 

Table J. Intraclass Correlations of Factor Ratings 
Across 99 Selected Job Titles* 





Factor 


Numbtr 
Of Ratings 


Intraclass 
Correlation 


1 


Formal Education 


15 


.92 


2 


Special Training and Work Experience 


9 


..96 


3 


Working Conditions 


15 


.92 


4 


Originality, Ingenuity, and Creativeness 


10 


.94 


5 


Communication Skills 


11 


.94 


6 


Interpersonal Skills 


11 


.94 


7 


Judgment and Decision Making 


11 


.95 


8 


Planning 


15 


.92 


9. 


Management 


9 


.95 


10 


Risk 


15 


.96 



Only factors 2, 5, 7, 8 and 9 arc wciglitcd into the OCR Policy Equation. Tlic remaining factor scales were developed 
for possible application in pay evaluation studies. 



Each set of 99 job titles as ordered by the mean ranking was sub-divided into 9 groups of 1 1 titles. 
The three jobs with ratings of smallest variance in each subgroup were chosen for inclusion in the list of 27 
titles composing the benchmark scale for each factor. When the 27 had been so identified, the derived scales 
were reviewed. In a number of cases, titles of slightly greater variance were substituted in order to represent 
more utilization fields in the scales. 

Although data for the job titles had been collected in a context of specific organizational 
identification and geographical address, when the final benchmark scales were prepared, the organizational 
identification was altered to indicate a type of organization, and the geographical address was either deleted 
or changed to a general designator, such as "overseas." An example benchmark scale is presented as Figure 
2. 

Collection of Job Evaluations Using the Benchmark Scale 

In order to test the efficiency of the benchmark scales, they were circulated to volunteer rating 
officers in the grade of major and lieutenant colonel, each of whom evaluated five job descriptions selected 
from the OCR criterion sample. One thousand such jobs were evaluated, and benchmark scale factor ratings 
were obtained for nearly all the jobs from not less than five raters. 

Predictive Efficiency of the Benchmark Scales 

Table 6 reports the correlations of benchmark scale factor ratings and grade ratings with the original 
OCR grade evaluations. They are generally as higli or higher than validities obtained using the OCR 
adjuective scales. The validity for an equation combining benchmark factor ratings with field grade ratings, 
supervisory grade ratings, and organizational factors was computed to be .90. This is slightly lower than the 
.92 reported in the original OGR study. The modest loss in validity is attributable to the grade ratings 
obtained from officers participating m the benchmark scale study, wliich produced less validity than those 
obtained from officers participating in the OGR study (.86 vs .89). 

Development of Integer Weight Equation 

Demonstration of the predictive efficiency of the benchmark scales supported a decision to develop a 
system suitable for hand application by local manpower personnel. First, an integer weiglit was developed 




17 



20 



FACTOR 9: MANAGEMENT: Tl)c level of executive, and njanagerial skills required in the job. 
Consider the complexity, variety, and level of the activities which are directed, organized, 
coordinated, controlled, commanded, or evaluated, 

LEVEL 9 

Diiector of Budget, Hq Major Air Command 
Commander, Combat Support Gp (Overseas) 
Wing Commander, Tactical Control Wg (Overseas) 

LEVELS 

Wing Commander, Aerospace Rescue & Recovery Wg 
Chief of Operations, Strategic Missfle Sq 
Deputy Commander, Air Base Gp 

LEVEL? 

Maintenance Supervisor, Avionics Maintenance Sq 
Squadron Operations Officer, Combat Crew Training Sq 
Base Accounting & Finance Officer, Rying Training Wg 

LEVEL 6 

Chief, Consolidated Base Personnel Office, Combat Support Gp 

Base Procurement Officer, Pilot Training Wg 

Helicopter Squadron Operations Officer, Rying Training Sq 

LEVELS 

Traffic Management Officer, Transportation Sq 

Base Communications Maintenance Officer, Communications Sq (Overseas) 
Missile Combat Crew Commander, Strategic Missile Sq 

LEVEL 4 

Chief, Utilities Operations Division, Civil Engineering Sq 
Chief, Photo Evaluation Branch, Photographic Sq 
Base Fu«?ls Officer, Supply Sq 

LEVEL 3 

Primary Pilot Training Instructor, Pilot Training Sq 
Space Smveillance Officer, Aerospace Support Sq 
Air Traffic Controller, Communication Sq 

LEVEL 2 

Administrative Officer, Air Base Sq 

Data Services Officer, Combat Support Gp 

Tactical Fighter Pilot, Tactical Fighter Sq 

LEVEL 1 

Clmical Psychologist, USAF Hospital 
Psychiatric Social Worker, USAF Hospital 
Helicopter Pilot Single Rotor, Air Base Sq 

Fig. 2, Example of benchmark scale, 
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Table 6. Correlations Between Mean Factor and 
Grade Ratings with OCR Criterion Board Action 



^'^^Q*^ Correlations 



1 Formal Education 61 

*2 Special Training and Work Experience .6 1 

3 Working Conditions _*03 

4 Originality, Ingenuity, and Creativeness ,67 
*5 Communication Skills j2 

6 Interpersonal Skills '69 

*7 Judgment and Decision Making [75 

*8 Planning j6 

*9 Management '79 

10 Risk _]22 

1 1 Mean Grade Rating 86 

♦Weighted into the grade evaluation composite. 

for each level on the supervisory grade rating scale, as shown in Table 7. This permitted the supervisory 
grade ratings to be treated as a single variable in the final integer^weight equation, which is presented in 
Table 8. The predictive vaHdity of this equation for the original Policy Board grade ratings was computed to 
be .90- 

Table 7. Integer Weights for Various Levels 
of Supervisor's Judged Grade 



Inttger Weight 

Supervisor's Judged Grade Lieutenant -3 
Supervisor's Judged Grade Captain -2 

1 
5 



Supervisor's Judged Grade Major 
Supervisor's Judged Grade Lieutenant Colonel 
Supervisor's Judged Grade Colonel 10 
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Table & Integer Weights for Final Grade 
Prediction Equation 



Integer Weight 

Rated Data From Field 

Special Training and Work Experience 3 
Communication Skills 4 
Judgment and Decision Making 2 
Planning I 

Management 12 

Mean Grade Rating from Field 4 
Organizational Level of Job 1 
Level of Job Within Organization 1 

Supervisor's Judgment Subcomposite 4 
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Summary of 1966 Benchmark Scale Study 

Benchmark Scales having rating levels defined by job titles were developed for factors entering into 
the 1964 OCR Policy Board grade equation. Tliesc scales were applied to a sample of 1,000 officer 
positions and were found to produce factor ratings having equal of lugher validities than those obtained 
with the original OCR adjective scales, a simplified integer-weiglit grade equation was developed for 
possible operational application to officer positions. This equation produced a validity of .90 for Policy 
Board grade decisions. 



V. OFFICER GRADE REQUIREMENTS RESEARGI, 1966-1974 

Tlius, by the end of 1966, a grade evaluation technology had been developed and partially tested 
which could be applied to individual officer positions. Only one more step was needed, and that was the 
construction of an equi-percentile conversion table. The prediction composite generated by the benchmark 
scale equation could be applied to order jobs from high to low in terms of merited grade. The conversion 
table was needed for converting composite score values into specific grade decisions. To put it another way, 
the conversion table is needed to make sure that the technology will assign the same distribution of grades 
to a set of positions as would have been allocated to that same set of positions by tlie 1964 Policy Board. 
Tlie equation is needed to make sure that the grades within this distribution are attached to the proper 
positions. Both devices are required to complete the technology. 

Be that as it may, the Air Force did not elect to implenwnt the OCR technology at that time, and the 
conversion table was not developed. The reasons why the OCR technology was not implemented are 
unknown, although some can be hypothesized. First, the Air Force was underway with "The Road Ahead" 
studies to determine the optimum distribution of the officer force. This culminated in TOPLINE, which 
calls for a specific grade distribution to support career objectives. Also, implementation of the OCR 
technology would have created problems in many utilization fields. The need for keeping pilots and 
navigators in the cockpit in order to amortize training investments has already been mentioned. Another 
example is found in the scientific and engineering area where the OCR study indicated that many positions 
being filled by lieutenants and captains should have been filled by majors. Yet, there was no way this 
requirement could have been met without attainment of a much higher retention rate for junior officers in 
this area. 

Regardless of the reasons, the Air Force elected not to implement the OCR teclmology, and research 
efforts were transferred from job evaluation to other programs. This condition held until 1974. 



VI. TEST APPLICATION OF OCR TECHNOLOGY BY MANAGEMENT ENGINEERING TEAMS 

In May 1974, AFHRL received a "Request for Personnel Research" (RPR 74-20), jointly sponsored 
by Hqs USAF PRMRE and DPXX, which asked for the development and testing of a "a technology by 
which management engineering teams can apply job evaluation factors to determine appropriate grade 
requirements for all officer positions, excluding those to be filled by line pilots and navigators, physicians, 
dentists, and personnel not subject to constraints of the Officer Grade Limitations Act (OGLA)." This 
research request was accepted, and work was initiated with a due date for completion in April 1975. The 
remaining portion of this report presents the status of the study as of 15 January 1975. 

Approach 

As indicated previously, two elements are needed for an acceptable grade evaluation system: (1) a 
"yardstick" which measures grade requirement levels, and (2) a method for reliably applying the yardstick 
to individual officer positions. If one does not trust the "yardstick," then he will not accept the final results 
- no matter how carefully that yardstick may have been applied. Fortunately, the Air Force had a 
yardstick (the OCR grade policy equation) on the shelf which was developed in the 1964 Officer Grade 
Requirements Project. A procedure for applying the yardstick to individual officer positions was developed 
in the Benchmark Scale study conducted in 1966. The only remaining requirement was for a procedure by 
which management engineering teams (MET) could reliably apply the yardstick to officer positions in the 
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current force. Whatever that procedure, it must be denwnstrated that the grades it allocates to positions are 
Uic same as would have been allocated by the 1964 OCR Policy Board. The policy expressed by tliat Board 
is the foundation of the proposed system. 

In order to tie the proposed system to the Board policy as directly as possible, the decision was made 
to have MET members provide factor ratings on a subset of positions from the original 3,575 criterion 
sample. Tliese ratings would be obtained using the benchmark scales developed in the 1966 study. The 
simplified integer weiglits developed in the 1966 study would be applied, and the resulting grade composite 
scores would be correlated witli the original Policy Board grade ratings. A high correlation would indicate 
that ratings obtained by tlie MET can be used to determine the appropriate grade requirements for 
individual officer positions. A decision was made to collect job descriptions and grade factor ratings on an 
additional sample of current jobs. Tliese data could be treated to yield a very rougli indication as to 
whether the procedure* would likely result in statements of grade requirments wliich are grossly divergent 
from those currently specified in UDLs. 

Selection of the Job Sample 

As indicated previously, two samples of officer positions were required by the study design. One was 
a sample of the positions originally rated by the Policy Board in 1964. Tlie second was a sample of 
positions which is representative of the total population of existing positions for which the teclmology is to 
be applicable. 

The first sample was selected by simply accepting all non-aircrew positions in the "1,000 case" 1966 
Benchmark Scale study which were deemed to still exist in the current Air Force inventory of jobs. This 
yielded a sample of 485 positions for which job descriptions. Policy Board grade ratings, 1964 field officer 
grade and factor ratings, and 1966 field officer grade and benchmark scale factor ratings were available. 

For the second sample, approximately 1,800 currently-occupied positions were identified which were 
judged to be representative of the total population of existing non-aircrew positions. This sample contained 
at least one position from each UDL grade-by-AFSC category, with supplementary positions being added 
from the more populated categories. 

Data Conection 

Eighty-nine n^anagement engineering teams participated in the data collection phase. Each MET was 
asked to collect position descriptions for a specified subset of the 1,800-job sample, while subsets of 
description^ from the 485-job sample were reproducted by AFHRL and distributed to each MET. Due to 
nonavailabiHty of some officer job incumbents, descriptions were finally generated for 1,687 of the 1,800 
job sample. 

An average of 14.93 MET members provided grade and benchmark factor ratings for each position in 
the 485 job sample. An average of 7.53 MET members provided such data for each position in the 1,687 

job sample. In all, 221 ,430 job-by-faclor ratings were obtained from 671 MET participants. 
Data Analyses 

Data analyses can be divided into two sets. The first set was designed to evaluate whetlier information 
collected by the MET produced a grade composite having a high correlation with the 1964 Policy Board 
grade ratings. These analyses were conducted on the 485-job sample and involved weighing together the 
grade and benchmark scale ratings provided by the MET, together v^th supervisory grade ratings and data 
conceming the organizational level of each position. Since the system was being evaluated for possible 
operational implementation, the simplified integer-weight equation developed in the 1966 Benchmark 
study was applied. The second set of analyses involved applying the equations to the 1,687 job sample and 
comparing the resulting equation grades with tliose currently held by job incumbents and also with tliose 
authorized on the UDL. 

Analysis of the 485 Job Sample 

Table 9 presents validity coefficients for each of the equation variables, as well as for tlie final grade 
evaluation composite. The composite vaHdity of .90 is essentially identical with that obtained in the 1966 
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Benchmark study, and is only slighly lower than the 92 obtained for the equation developed in the 1964 
OGR study. At least part of the loss from .92 to .90 was due to the use of simplified integer-weigfits rather 
than least-square weigfits. A sixteen-poiiit equi-percentile conversion table (Appendix C) was constructed 
and applied to the grade equation composites to eliminate tlie restriction in range due to regression effects. 
Figure 3 was constructed to display the rclationsliip between the grades assigned by the equation and those 
assigned by the 1964 Policy Board to these 485 positions. It will be recalled that three points on the scale 
♦ encompasses one grade level (except for lieutenants). Thus, for tliose positions in the diagonal squares, 

there was exact agreement between the equation and board ratings. Positions plotted one square off the 
diagonal are those for which the equation and board agreed witliin one-third of a grade level. Positions 
plotted two squares off the diagonal are those for which agreement was witliin two-thirds of a grade level. 
Data in the figure reveal that for 94% of the positions, agreement was perfect or witliin two-thirds of a 
grade, and for only one percent of jobs was there disagreement by nK>re than one full grade level, in 
summary, it appears that the MET can accurately implement the 1964 criterion board policy, using the 
benchmark scale factors and the simplified integer weight equation. 



Table 9. VtKditks of Variables Induded in the 
MET-AppUcd Integer Weight Grade Equation 



Variables 


VaMditiM 


Factor 2, Special Training and Work Experience 


.61 


Factor 5 Communication Skills 


.72 


Factor 7 Judgment and Decision Making 


.73 


Factor 8 Planning 


.78 


Factor 9 Management 


.82 


Level of Organization 


.48 


Level of Job Within Organization 


.49 


Supv Judgment of Grade for Job 


.79 


Mean MET Judges Grade Ratings 


.86 


Final Grade Evaluation Composite R = .90 





If the technology were to be implemented operationally by MET, it is estimated that the validity of 
the grade equation would be somewhere near the .90 reported previously. However, this estimate is 
obtained by balancing a number of complex considerations into a single judgment. First, the mean grade 
and job factor ratings used in the test validation were based upon an average of nearly fifteen raters. It 
would be expensive for the Air Force to routinely involve fifteen raters in application of the technology to 
every officer position. Use of five raters would be a reasoaabk recommendation. Means based upon five 
raters are less stable than means based upon fifteen raters, and this reduction in stability would result in 
grade composites with less validity. Prophecy formulas (Guilford, 1956, p. 475) indicate that if only five 
raters had been used in the present study, the resultant validity would have been approximately .85 in the 
485 case sample. This is high, but nevertheless is significantly lower than the reported .90, The reliability 
coefficient for the fmal composite based on five raters in the 485 job sample is computed to be .91. 
However, the reliability coefTicient for the grade composite based on five raters in the 1,687 case sample 
was computed to be .94. This higher value was probably due to tlie fact that MET members were more 
familiar with the jobs in the latter sample. In this instance, they were rating jobs in their own command and 
located at or near their own installation. By accepting certain tentable assumptions, it can be estimated that 
for officer positions serviced by the MET, the validity based upon five raters would be at least .87, Finally, 
some error variance was introducted into the study data by raters who failed to follow instructions, either 
through misunderstanding or lack appreciation of the need for preciseness. In an operational system, tliis 
variance could be eliminated or greatly reduced by review procedures. To summarize, it is estimated that 
grade evaluation compoates based upon five MET raters would have a validity of approximately ,90 in a 
^ carefully monitored operational system. 

Validity in an operational system could be further enhanced by making maximum use of officer 
raters, Fo the 485 job sample, the ratings provided by civilian MET members were somewhat more rehablc 
than those provided by airmen MET members, while the ratings provided by officer MET members were 
more reliable than those provided by either airmen or civilians. Raters in the test study consisted of 374 
enlisted, 130 civilian and 167 officer personnel. 
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Fig. 3. Comparison of grades allocated to 485 jobs by the 1964 Policy Board and the MET applied 

grade evaluation equation. 
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As mentioned previously, an equi-percentile conversion table was developed so that equation grade 
evaluation composite scores could be converted into statements of grade requirements. Table 10 presents 
data comparing distributions of UMD grade allocations, Policy Board grade judgments, converted equation 
grade evaluations, and the actual grades held by officers in these 485 positions in 1964. Distributions of 
grades allocated by the equation and by the Policy Board are essentially identical. This is as it should be, 
since the conversion table was designed to bring the two distributions into alignment. It can be seen that 
the equation (and Policy Board) allocated about the same number of colonel positions as the number of 
colonels available in the 485-case sample. The equation called for a few more lieutenant colonel positions 
and a much larger number of major positions than reflected either in the UMD or held by the on-board 
population of officers. This is consistent with the general finding in the original 1964 OCR study that many 
Air Force jobs filled by captains and lieutenants should, in fact, be filled by-jnajors. 

Table 10. Comparison of Grade Distributions 
for the 485 Job Sample 



Grad« 




Equation 


UMD 


Prwant 


Lfvd 


Board 


Gradf 


Gracta 


Grad* 


Colonel 


48 


48 


48 


45 


Lt Colonel 


89 


89 


80- 


80 


Major 


178 


177 


126 


121 


Capt/Lt 


170 


171 


231 


239 


Total 


485 


485 


485 


485 



Analysis of the 1 ,687 Job Sample 

Grade evaluation composite scores were computed for each position in the 1,687 job sample. These 
composite scores were then converted into estimated grade requirements using the equi-perccntile 
conversion table developed in the 485 job sample. Table 1 1 presents the resultuig distribution of equation 
grade requirements, along with the UDL requirements and grades currently held by officers in these 
positions. 

Table IL Comparison of Grade Distributions 
for the 1,687 Job Sample 



Gradt 


Equation 


UDL 


Prtsant 




Grade 


Grad* 


Gradt 


Colonel 


115 


121 


108 


Lt Colonel 


271 


307 


275 


Major 


604 


420 


415 


Capt/Lt 


697 


799 


889 


Total 


1,687 


1,647* 


1,687 



*Plus 40 overages. 

A degree of caution should be exercised in interpreting the equation grade requirements in Table 11. 
The 485 job sample was adequate for determining the relationship between the grade evaluation composite 
distribution and grades assigned by the 1964 Policy Board. However, this sample is considered to be too 
small to construct a stable conversion table. This means that the cutting points identified in the composite 
distribution to separate adjacent grades contain too much margin of error. Before the grade evaluation 
technology can be implemented operationally by MET, it will be necessary to obtain ratings on an 
additional 500 to 1,000 positions from the original 1964 criterion sample in order to construct a stable 
conversion table. 

The data in Table 10 reveal apparent inflation in the UDL at the colonel and lieutenant colonel levels, 
and an understatement by the UDL for requirements at the major level. The equation allocates a few more 
colonel and slightly fewer Heutenant colonel positions than held by the Lncumbents in the 1,687 job 
sample. Again, the data reflect the need for more majors. 
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It is dangerous to generalize from 1,687 positions to the entire officer job population, especially 
knowing that the grade composite conversion table was constructed using only 485 cases. Yet, if one had to 
hazard a guess as to what the technology might yield for the total non-aircrew force, it would be most 
reasonable for one to assunoe that the trends iii Table 10 hold true. 

When one compares data in Table 10 with those in Table 9, there appears to be an inconsistency. In 
the 485 job sample, the number of equation grades at the colonel and lieutenant colonel levels is equal to or 
higher than the UMD. Yet in the 1 ,687 job sample, the number of equation grades is lower than the UDL at 
both of these levels.^ An analysis was made to see if this inconsistency was due to biases in the job samples 
or inflation in authorized grade distributions. Data presented in Table 12 suggest that there has been a 
signiflcant inflation in the levels of grades authorized for non-aircrew positions since 1964. 

Table 12. ComparBonsof 1964 and 1974 UMD/UDL 
Grade Authorizations for Non-Aircrew Positions' 



19€4 1974 



Grade 
Lavtl 


Fr«q 


% 


Fr#q 




Colonel 


3,859 


5.31 


5,341 


8.03 


Lt Colonel 


9,278 


12.77 


11,588 


17.44 


Major 


16,454 


22.64 


16,839 


25.34 


Lt/Capt 


43,084 


59.28 


32,673 


49.18 


Total 


72,675 




66,441 





^Excludes Aircrew personnel (lOXX, IIXX, V2XX, 13XX, 15XX), Spec Inv 
(82XX). Physicians and Dentists (93XX, 94XX, 95XX, 96XX, 98XX),aJid special 
duty (000X,01XX,09XX). 



Vn. DISCUSSION OF RESULTS 

Based upon the results presented, it appears that management engineering teams can evaluate grade 
requirements for non-aircrew officer positions through application of the benchmark scales and the 
integer-weight equation from the 1966 study. In effect, this procedure will assign the same grade level to 
each position as would have been assigned by the 1964 Policy Board. Thus, any inflationary trend in grade 
allocations which may have occurred since 1964 will be eliminated by implementation of the procedure. 

The question might arise whether a new board should be established to express grade policy as of 
1975. After all, many new types of officer positions have been introduced since 1964, and the original 
equation may not be valid for such positions. 

There are many reasons for sticking with the 1964 board's policy statements. The Officer Grade 
Requirements Project was one of the largest and most-care fully executed job evaluation studies ever 
accomplished - in or jDut of the military services. Numerous analyses were conducted over a one-year 
period before the board judgments were accepted as a criterion. Ratings provided by board members were 
demonstrated to be reliable, reasonable, and unbiased. The job factors which were identified for replicating 
the board's judgments have high face validity for defining the meaning of grade. Every step in the project 
wab carefully documented, and all raw data and analyses runs are still available for inspection. An 
atmosphere was created during the board rating sessions which would be difficult to reproduce in today's 
environment. 



^In 1964, aathonzcd grades were recorded on Unit Manning Documents (UMD). In 1974, grade authorizations were 
posted on Unit Document Listings (UDL). In the present report, UMD can be treated as the equivalent of UDL. 
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Furthermore, there is every reason to believe that the original equation is as valid for today's jobs as it 
was for jobs existing in 1964. During the 1964 study, tests were conducted to determine if different factors 
and weights were needed for expressing the board's grade policy for different classes of jobs. In 1974 data 
was rechecked across Ave job sets using restricted and full sets of variables, and no significant increases in 
ihe multiple regression coefficient were found. This implied that the factor variables selected in the final 
regression equation provided optimal predictive efficiency. It appears that grade has a universal meaning, 
and that the same factors and weights are applicable to all classes of jobs. There is no reason that the same 
equation should not be valid for any new types of job which have come into the inventory since that time. 

Finally, tying grade requirements to a constant standard mil protect the Air Force from any 
unjustified **grade creep/' Assuming honest api^cation of the benchmark scales, any increase in stated 
grade requirements will be associated with actual increased responsibility levels associated with jobs. 



Vni. SUMMARY AND CONCLUSIONS 

This report has described the development and test of a system by which Air Force management 
engineering teams (MET) can evaluate grade requirements for individual officer positions based upon job 
content and responsibflity levels. The **yardstick'' used for grade evaluation was developed in the 1964 
Officer Grade Requirements (OCR) Project -one of the largest and best documented job evaluation 
studies ever accomplished. During the 1964 project, twenty-two highly experienced colonels provided grade 
requirement ratings for 3,575 positions which were carefufly selected to represent nearly all types and levels 
of ofiicer jobs in the Air Force. Subsequent analyses indicated that these "Policy Board" colonels had high 
interrater agreement concerning the appropriate grades for positions in the 3,575 job sample; that they did 
not display biases toward jobs in individual specialties or commands; that they did not produce inflated 
grade estimates; that they did not simply confirm existing authorized grades; and that they were confident 
in their judgments. It appeared that each job had been judged on its individual merits; so the board ratings 
were accepted as the basis for constructing a job evaluation system. 

An equation system was developed, using the policy-capturing model, which was demonstrated to 
assign the same grade levels to officer positions as the Policy Board (validity = .92). It weighted together 
factors with high face validity for grade, including those measuring job requirements for managerial skUls, 
planning, communication skills, judgment, decision making, special training, work experience and the 
organizational level of the job. 

A follow-on study was conducted in 1966 in which "benchmark scales" were developed for use in 
evaluating individual officer positions in terms of requirement levels on factors entering into .the policy 
equation. The present report summarizes the results of a test to see if management engineering teams can 
apply these benchmark scales and accurately determine grade requirements. 

A total of 671 MET members participated in the test. They provided 221,430 factor-by-job ratings 
which made possible the computation of grade evaluation composites for 485 officer positions previously 
rated by the 1964 Policy Board, and deemed still to exist in the Air Force inventory. They also rated 1 ,687 
current positions selected to be representative of the existing Air Force population of non-aircrew jobs. 
Analyses of the 485 job sample indicated that management engineering teams can accurately implement the 
1964 board's grade "policy.'' Analyses of the 1,687 job sample indicated that if the system were 
implemented operationally it would probably reflect the need for fewer colonel and lieutenant colonel 
positions than currently authorized on the UDL, but about the same number as held by the present 
on-board officer population. Again projecting from the 1,687 job sample, the nev/ system would probably 
indicate the need for more major positions than currently authorized. These projections are extremely 
tenuous because of the small base sample (1,687), and the weak conversion table, based only on 485 job 
samples, used for translating grade composite scores into grade decisions. 



IX RECOMMENDATIONS 

If the Air Force management wishes to implement a system for evaluating grade requirements based 
upon job content and responsibilities, it is recommended that the technology described in this report be 
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adopted. It is backed by years of systematically-accomplislied research and, considering the present 
state-of-the-art, it is the most defensible system likely to become available in the foreseeable future. 

Before the technology becomes operational, it will be necessary for management engineering teams to 
evaluate another 500 to 1,000 officer positions from the original 1964 job criterion sample. These 
evaluation would be used to construct a stable conversion table for translating grade composite scores into 
grade decisions. 

If the Air Force needs to determine the total distribution of non-aircrew grade requirements for 
planning purposes, evaluations on 10,000 to 15,000 officer positions by MET would be needed. Projections 
could be made from this base to determine the grade requirements for each non-aircrew utilization field. 

It is recognized that certain grade authorizations arc required for the Air Force to meet career 
plarming objectives. It is recommended that the requirements based upon job demands be compared with 
those needed to support career planning objectives in order to determine the best grade structure to service 
both needs. 
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APPENDIX A. DEFINITION OF JOB REQUIREMENT FACTORS 



1. FORMAL EDUCATION: The amount of formal education required by the Job. Consider the 
education obtained in high school, college, university, or professional school. 

2. SPECIAL TRAINING AND WORK EXPERIENCE: The extent to which the Job requires 
knowledges and skills which must be acquired through special training courses or on-the-job 
experience. Disregard general courses given by Squadron Officer School, Command and Staff 
College, or War College. 

3. WORKING CONDITIONS: The extent to which the Job involves uncomfortable working condi- 
tions. Consider such conditions as isolation, irregular hours, monotony, prolonged vigilance, 
extensive TDY, and pressure to meet deadlines. 

4. ORIGINALITY, INGENUITY, AND CREATIVENESS: The extent to which the Job requires 
new and unique methods, approaches, and solutions to problems. Consider the demand for 
novel ideas and inventiveness. 

5. COMMUNICATION SKILLS: The extent to which the Job requires skill in oral and written 
communication. Consider the complexity and variety of information communicated as well 
as the level of the individuals and agencies involved. 

6. INTERPERSONAL SKILLS: The extent to which the Job requires skill in dealing with people. 
Consider the need for sensitiveness, responsiveness, persuasiveness, self control, and tact, 
as well as the possible consequences when such skills are not employed. 

7. JUDGMENT AND DECISION MAKING: The importance and independence of Judgments and 
decisions required by the job. Consider the nature, variety, and possible impact of decisions. 
The less well defined the guidance for decisions, the higher should be the rating; while the 
more specific and detailed the guidance, the lower should be the rating. 

8. PLANNING: The extent to which planning is required by the Job. Consider the scope and 
significance of work for which planning is done. The longer the time span for which plan- 
ning is done, the higher the rating should be. 

9. MANAGEMENT: The level of executive, and managerial skills required in the Job. Consider 
the complexity, variety, and level of the activities which are directed, organized, coordinated, 
controlled, commanded, or evaluated. 

10. RISK: The extent to which the Job requires exposure to risk of death or severe injury in 
peace-time. 
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APPENDIX h. PARTIAL LIST OF PREDICTIONS EVALUATED DURING DEVELOPMENT 

OF THE OGR POLICY 



Predictor 



Definition 



Type of Score 



'4 

4 



no 
^11 

^12 
^13 

^16 
^17 

Xi8 

^19 
^20 
^21 
*22 
X23 

^26 
^27 
^28 

J30 
pl 

^33 



Formal education 

Special training and work experience 
Working conditions 

Originality, ingenuity and creativeness 

Communication skills 

Interpersonal skills 

Judgment and decision making 

Planning 

Management 

Risk 

Grade rating from field judges 
Pilot job 

Commander or director job 

Navigator job 

Air operations job 

Scientific or development engineering job 
Electronic, maintenance, or civil engineering 
job 

Materiel or comptroller job 
Administrative or support job 
Professional job 
Level of organization 
1 if X2i= 0; 0 otherwise 
^21 squared 

Level within .^organization 
1 if 7.21^- 0; 0 otherwise 
Xgj^ squared 

Supervisor's judgment of grade 



27 

28 
)2Q 



Lieutenant 
Captain 
Major 
Lt Colonel 
Colonel 
General 



28 

X28 squared 
UMD authorized grade 
squared 
= Lieutenant 
= Captain 
= Major 
= Lt Colonel 
= Colonel 



(35 

35 

35 

35 
^35 



continuous 
continuous 
continuous 
continuous 
continuous 
continuous 
continuous 
continuous 
continuous 
continuous 
continuous 
1 or 0 



or 
or 
or 
or 



1 or 0 
1 or 0 
1 or 0 
1 or 0 
continuous 
1 or 0 
continuous 
continuous 
1 or 0 
continuous 
continuous 
1 or 0 



or 
or 
or 
or 
or 



continuous 
continuous 
continuous 
1 or 0 



or 
or 
or 
or 
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Appendix B. (Continued) 



Predictor 



Definition 



Type of Score 



\2 
^k3 
\k 

lk3 

Jut 

JU8 

51 
X52 

^55 
41 

^59 
^60 
^61 
^62 

^63 
^6k 
65 

X6T 

J68 
J69 

X71 
JT2 

hk 
75 

hQ 
J83 

^85 



Inctimbent's present grade 
Xij2 squared 

= Lieutenant 
= Captain 
= Major 
= Lt Colonel 
= Colonel 

s grade higher than 



^1»2 

Incumbent ' 
Incumbent ' s 
Incumbent • s 



UMD 



grade = to U1«ID 
grade lower than 
Grade of immediate supervisor 



UMD 



X 



^52 
52 



squared 
= civilian 
Lieutenant 
Captain 
Major 
Lt Colonel 
Colonel 
General 



52 . 

?2 = 
= 

^52 = 
52 

Supervisor's grade higher than incumbent 
Supervisor's grade equal to incumbent 
Job is in SAC 
Job is in TAC 

Job is in PACAF, USAFE, or USAFSS 
Job is in Hq Command or Hq USAF 
Job is in AFSC or OAR 
Job is in AFLC, AFAF'C, or AFCS 
Job is in AAC, SOU, ADC, or CONAC 
Job is in ATC, AU, or USAFA 
Job is in commands other than above 
Command and staff job 
Non-command and staff job 



•x 



squared 
squared 
squared 
squared 
squared 
squared 
squared 
squared 
squared 



(squared job evaluation 
factor scores; introduced 
to test for non-linearity) 



^10 s^^^^^^^ 
^11 



continuous 
continuous 
1 or 0 



or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 



continuous 
continuous- 
1 or 0 



or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 
or 0 



= 1 
= 3 



or 2^ 



1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

continuous 

continuous 

continuous 

continuous 

continuous 

continuous 

continuous 

continuous 

continuous 

continuous 

1 or 0 

1 or 0 
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Appendix (Continued) 



Predictor 



Definition 



Type of Score 



086 

^88 
X89 

bo 

091 
p2 

xP 

y9 

XlOO 
^101 

1102 

*io6 
%07 
Jl08 

b.09 
b.10 
nil 
JJ112 

ni3 

0115 
X116 

^117 
JJ118 
O119 
O120 
O121 
0122 
0123 
O12U 
*125 
X126 

Jl27 
0128 
^129 



^11 ~ 

^: 

^11 2 
^11 " 

^11 _ 

^11 2 

^11 " 

^11 

^11 ~ 
X = 

p = 

Oil _ 

021 - 

X21 = 

^21 ^ 

y2i : 
021 : 

41 = 

X214 - 
X214 = 
X214 = 

^21+ 7 
Xgl 

y22 
X2I4 

y25 
021 

^21 
X22 

y21 
X22 

X21 

y22 

21 
X22 

^22 

y21 
022 
X2I, 
X25 



1» 
5 
6 

7 
8 

9 

10 

11 
12 
13 
Ih 

15 or 
7 
6 
5 
h 
3 
2 

7 
6 
5 
1+ 
3 
2 

^13 
^13 

ijis 

O13 

^15 

Xi6 
^17 
^17 
^18 
^18 
^19 
^19 
X20 

^20 
X98 
X98 



(Variables associated 
with specific grade 
rating levels) 



16 



(Variables associated 
with specific levels 
of organization) 



(Variables associated 
with specific levels 
of organization) 



(Variables expressing 
interactions between 
level of organization 
and membership in 
various AFSCs) 



\ 



1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 
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Appendix B. (Continued) 



Predictor 



Definition 



Type of Score 



^130 
^131 
h.32 
Jl33 
^13h 
Jl35 
^136 

Jl38 

hko 
Hkl 

%5 
b.h6 
nU7 
^1U8 
^lU9 
^150 

^151 
tl52 
Jl53 
^15U 
*155 
^156 
Jl57 
^158 

^163 

Jl65 
^166 
Xi67 
Xi68 
X169 
^170 



J2l| 

X25 

^2 It 
X25 

X25 
^25 



Xqq) 

^100 { 
^101^ 

^102) 

X102) 

^103^ 



(Variables expressing 
interactions between 
level of organization 
and level within 
organization) 



Highest education 
Highest education 
Highest education 
Highest education 
Highest education 
Highest education 
Highest education 
Highest education 
Highest education 
Major field of study 
Major field of study 
Major field of study 
Major field of study 
Major field of study 
Major field of study 
Major field of study 
Major field of study 
Major field of study 
Aircrew operations job 
Non-aircrew operations job 
Scientific and engineering job 
Material and comptroller job 
Professional job 
Administrative and support job 
Missile associated job 
"Low glamour" job 
Length of job description 
Length of job req.uirement statements 
Total months in DAFSC 
Months in present assignment 
Months in present grade 



Non high school 
high school 

1 year college 

2 year college 

3 year college 
BA 

BA+ 

MA 

PhD 

Ed 
PE 

Arts & His 
Eng 
Bus 

Med, Leg 
Soc St 
Mil Sci 
other 



1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 


1 


or 


0 



continuous 
continuous 
continuous 
continuous 
continuous 
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APPENDIX C ILLUSTRATION OF A JOB DESCRIPTION AND RATINGS 



This appendix illustrates an actual job description from the 1,687 case sample and the MET ratings 
obtained on this description. The job (number 990531) was randomly selected. Data concerning application 
of the policy equation is also provided. 

As shown on pages 2 and 3 of the AF Officer Job Survey (Figure CI), the title of the job was 
"Aircraft Maintenance Supervisor" and it is located in an organizational maintenance squadron. Eight 
duties with tasks for each duty are listed. Examination of this information reveals specific details 
concerning job content and responsibilities. Additionally, a section covering unusual job requirements and a 
brief job summary are provided. 

Page 4 of the AF Officer Job Survey provides other assignment infonnation concerning the job, with 
certain personal data withheld to safeguard confidentiality. With regard to elements in the policy equation, 
the level of organization was a squadron (level 3), level of job was in some division within the squadron 
(level 5). Presumably the "division" job level is appropriate considering the job content (pp. 2 and 3), 
although squadrons typically are divided into flights, sections or detachments. The immediate supervisor's 
judged appropriate grade for the job was a major (level 4). 

Using only the information from pages 2 and 3, seven MET raters provided benchmark factor ratings 
(9-level scale) and grade ratings (16-point scale). Of the seven raters, five were enlisted and two were 
officers. The mean factor scores and mean field judges grade ratings, along with other information necessary 
to apply the grade equation to job #990531 are given in Table CI. This table includes the integer weights 
for each element of the equation and a cumulative composite score as it may be computed on a desk 
calculator. 

Table CL Application of Policy Equation to Job #990531 



Eitmtnt 


Score 


,Tlm«) Wttght 


CumuUtlvt 
= Scorm 


Factor 2 Special Training and Work Experience 


3.1 


3 


9.3 


Factor 5 Communication Skills 


3.6 


4 


23.7 


Factor 7 Judgment and Decision Making 


4.1 


2 


31.9 


Factor 8 Planning 


43 


1 


36.2 


Factor 9 Management 


5.0 


12 


96.2 


Mean Field Judges Grade Ratings 


6.1 


4 


120.6 


Level of Job 


5 


1 


125.6 


Level of Organization 


3 


1 


128.6 


Recoded Supv Judgment of Grade 


(4)=1» 


4 


132.6 


Grade Evaluation Composite 






132.6* 



'''Converts to Pay Grade = 7 (junior level major) on 16-pt grade conversion table or grade 4 (major on a 7-pt conversion 
table (Table 2). 

^Scc explanations in text concerning Table 7, and Brokaw and Giorgia (1966 study). 



As shown in Table CI, the final grade evaluation composite score was 132.6. Using the equi-percentile 
conversion scales (Table C2) developed on the 485 MET rated job sample, this value convene to a pay grade 
of 7 (a junior level of experience major). In comparison, the UDL grade for this position is level 4 (major), 
the present grade of the officer is level 3 (captain) and the supervisor's judged appropriate grade for the 
position is level 4 (major). 
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Table C2. Grade Conversion Table* 



Wii^htid Compositi 
CumuUtlvi Scori 


Conwsrtt Pav 
Gradi (IC'Pt Seal!) 


Convirts to Pay 
Gradi (7>pt Scale) 


Sec Text and Below^ 


16 General 


7 Cififipral 


289.1 or above 






961 1 to 780 0 


i*r ivjiuuie ^oionei 


0 Lolonel 


738 8 to 261 0 


1 J J uiijur i^oionei 




217 3 to 738 7 


i£i ociuur i^i ^oionei 




700 9 to 717 7 


I 1 MwlHIo T f P<n,l«nal 

I I iviiuuie Lii ^oioiiei 


J Lt Colonel 


183 5 to 200 8 


i\j J uiuui 1^1 v«.oionei 




162 5 to 183 4 


y Oviiiui iviajur 




143 9 to 167 4 


0 MiaQie iviajor 


4 Major 


-^122.1 to 143.8 


7 Junior Major 


105.3 to 122.6 


6 Senior Captain 




80.3 to 105.2 


5 Middle Captain 


3 Captain 


65.5 to 80.2 


4 Junior Captain 


52.3 to 64.5 


3 Senior Lieutenant 




52.2 and below^ 


2 Lieutenant 


2 Lieutenant 



Tliis tabic was based on data from the 485 case sample described in text of 
the staff report, and is not recommended for operational use. Additional research is 
required to produce a more stable. 



Although supervisory and MET grade ratings at the general level are allow- 
able, the conversion table does not recognize grade requirements above colonel. 

^At the present time, the system is not designed to distinguish between 
middle and junior lieutenant positions (levels 2 and 1). 



35 

38 



AIR FORCE OFFICER JOB SURVEY 



INSTRUCTIONS 

This survey ts directed by Hq USAF to identify and describe the work performed by officers In the Air Force. The 
Air Force needs precise information about the duties, tasks, and requirements of officer jobs in order to maintain the 
cidssificdtion structure, to make appropriate grade allocations, to define incumbent qualifications, and to guide other 
manpower and personnel actions. Participation in this survey gives you an opportunity to provide accurate information 
about your job In support of improved Air Force management. 

You are requested to complete the survey according to the following instructions: 

1. ASSIGNMENT INFORMATION (Page 4). Fill in the required data or check the one box in each block that 
applies to you. 

2. JOB DESCRIPTION (Pages 2 and 3), On these pages provide typewritten* information which accurately and 
comprehensively describes your job. 

a. in the JOB NAME OR TITLE block, record a name or title which is descriptive of your job. 

b. In the JOB CONTEXT block, locate your job within the organizational structure. 

Examples: (1) THIS JOB IS IN THE HEAVY EQUIPMENT BRANCH DIRECTLY UNDER 
THE BASE MOTOR POOL COMMANDER, WHO REPORTS TO THE M & S 
GROUP COMMANDEa 

(2) THIS JOB IS IN THE TARGETS SECTION OF THE OPERATIONS 
PLANNING BRANCH OF WING Ha 

c. In the blocks under DUTIES AND TASKS, list statements that describe your job. Consider significant work 
activities such as those involved in commanding, planning, organizing, directir)g, rTX)nitoring, coordinating, 
reviewing, inspecting, evaluating, supervising, and operating. Use as many blocks as you consider necessary. 
The statements you provide should clearly define your job. 

ExiimpW; Duty A. DIRECTING MATERIEL CONTROL FUNCTIONS 
TASKS (1) ASSIGN PRIORITIES TO REQUISITIONS 

(2) COORDINATE REQUIREMENTS FOR MOBILITY DEPLOYMENT 

(3) MONITOR SUPPLY BUDGET 

(4) PROCESS REQUESTS FOR LOCAL MANUFACTURE OF ITEMS 
<5) REQUISITION TIME CHANGE ITEMS 

First, list* all the major duties you perform then go back and list the appropriate tasks under each duty. 
Describe your normal iob. Omit temporary variations in your work which are not pari of your regular 
assignment. Ignore ac'dit'onal duties unless they constitute a significant part of your job. 

d. In the JOB REQUIREMENTS block, enter additional statements that describe any unusual requirements of 
your job for the factors below. 

COMMUNICATION SKILLS ORIGINALITY, INGENUITY, & CREATIVENESS MANAGEMENT 

INTERPERSONAL SKILLS SPECIAL TRAINING & WORK EXPERIENCE PLANNING 

WORKING CONDITIONS JUDGMENT & DECISION MAKING RISK 

FORMAL EDUCATION 

Examptos: (1) WORKING CONDITIONS: JOB REQUIRES APPROXIMATELY 
120 DAYS TOY ANNUALLY. 

(2) SPECIAL TRAINING & WORK EXPERIENCE; JOB REQUIRES A SO-DAY AF 
COURSE IN SPECIAL WEAPONS DELIVERY. 

e. In the JOB SUMMARY Block, write a three-or four«9entence summary description of your job. 

f. After you have coii'Dloted pages 2, 3, and 4, sign in the space provided on page 4 and submit this form to your 
supervisor. 

NOTE Supervisor will review all entries, check a box to indicate his judgment of the most appropriate grade level for 
this job, and sign the form. 



"M typing fl«rvlc« Is not avaiUbto, Information should b«cl««rly printed by.htnd. 



p^ppj FORM 80-00(M)17 

s«p74 fig, CL EximpSe of completed job description. 
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JOB DESCRIPTION 



JOB NAM6 OR TITLE 

Aircraft Maintenance Supervisor 

JOB CONTEXT ' 

This job is in the Organizational Maintenance Squadron directly under the Chief 
of Maintenance who reports to the Wing Commander. 



DUTIES AND TASKS 

DUTY A- Directing the Flightline Maintenance Effort 

Tasks 

(1) Translates broad management objectives to 4 officers, A branch chiefs and 
one senior NCO. 

(2) Assess the progress of maintenance completion on assigned aircraft. 

(3) Prpvides squadron inputs to Chief of Maintenance Staff. 
(A) Analyzes delays in schedule takeoff time. 

(5) Insure proper number of personnel are available at the right Lime. 



uutyb: Evaluate Quality of Work Performed by 404 Personnel 

Tasks 

(1) Review Quality Control MSEP reports. 

(2) Spot check maintenance performed by technicians. 

(3) Discuss problem areas with Quality Control. 
(A) Arrange for required training if necessary. 
(5) Demonstrates new ideas or methods. 



Coordinate Maintenance Activities With Other Squadrons 

Tasks 

(1) Discuss maintenance problems with counterparts. 

(2) Arranges for specialized support for flying schedule. 

(3) Arrange for transfer of equipment. 

(4) Organize meetings. 



Management of 4 Officers and 400 Maintenance Technicians 

Tasks 

(1) Insure assigned personnel have required clothing, equipment, tools and 
facilities. 

(2) Make spot checks to insure required clothing, equipment and tools are in 
use. 

(3) Discuss and provide solutions to personal problems. 

(4) Arrange for TDY of assigned personnel. 
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Monitor, Review and Supervise the Squadron Financial Budget 

TMks 

(1) Review the daily spending. 

(2) Determines what supplies are necessary and what supplies can be eliminated 

(3) Provide inputs to yearly budget. 

(4) Review requests for all money expenditures in squadron. 



OUTYF: Assumes Squadron Commander's Responsibilities During His Absence 



OUTYG: Control and Supervise the Operation and Maintenance of: 
Tmk% 

(1) 68 assigned aircraft and installation equipment worth over $1 1 1 ,C 00,000 . 

(2) 52 vehicles worth over $614,000. 

(3) 15 trailers worth over $45,000. 



D^YH: ^g^intain Responsibility for Safeguarding and Condition of: 
Tasks 

(1) 5 buildings, aircraft parking ramp and surrounding areas. 

(2) Tools worth over $15,000. 

(3) Aircraft parts worth many thousand dollars. 

(4) Office equipment worth over $5,000. 



JOB REQUIREMENTS 

Working Conditions : Job requires possible TDY on short notice. Amount of TDY is 
dependent on world political stability. Working hours up to 24 hours per day. On 
call 24 hours a day. Shift work other than 8-4 shift on occasions. Subject to 
work outdoors under adverse conditions. 

Special Training and Work Experience : Job requires minimum of 6 week course in 
USAF Aircraft Maintenance. Must be familiar with unit an^ SAC mission as well as 
SAC exercises, emergency war orders, and contingencies. 

Communicative Skills : Required to brief aircraft status, accidents and incidents, 
and plans. 



JOB SUMMARY 

This job entails managing assigned personnel equipment, and facilities to 
accomplish the maintenance workload in a safe and timely manner. This is 
accomplished by translating the broad management objectives of the chief of 
maintenance into specific responsibilities of the branch chiefs. 



AF OFFICER JOB SURVEY Page 3 

Fig. CI {Continued) 



'41 



990531 

(1:1.4) 



ASSIGNMENT INFORMATION 



LAST NAME 

WITHHELD 



FIRST NAME 



M.I. 



ORGANIZATION (5.40) 

509th OMS 



BASE OR INSTALLATION (41.80) 

Pease AFBf 



SOCIAL SECURrTY ACCOUNT NUMBER 



YOUR PRIMARY AFSC 







\ 


/IT] 


IHEl 











Numbtr 

(2:5-13) 



Suffix 
(14«15) 



□ 

Prefix 
(16) 



4 


0 


2 





Number 
(17»20) 



□ 

Suffix 
(21) 



UMO AUTHORIZEO 
GRAOE FOR YOUR 
POSITION 

(22) 

Overage A 
W/O Q B 

Lt □ 2 

CaPt Q 3 

Mat Q 4 

L/C □ 5 

Col Q 6 



LEVEL OF YOUR 
ORGANIZATION 



OOO^ 
Hq USAF 

Hq Maj Air 

Co md 

Numbered AF 
(or Equlv) 
Air Division 
(or Equlv) 

(or Equiv) 
Group 
(or Equiv) 

Squadron 
(or Equiv) 

Detachment 
(or Equiv) 

Other (Specify) 



(39) 

□ 5 

d\ 

□ 7 

□ 6 

□ 2 



YOUR PRESENT 
GRAOE 



W/O 

2/Lt 

1/Lt 

CaPt 

Ma] 

L/C 

Col 



(23) 

□ » 



□ 1 

□ * 



GRAOE OF YOUR 
IMMEOIATE 
SUPERVISOR 



YOUR OUTY AFSC 



Civ 

2/Lt 

1/Lt 

CaPt 

Mai 

L/C 

Col 

Gen 



(24) 

□ « 

□ 1 

□ 2 

□ 3 

□ ^ 

□ « 

□ 7 



□ 

Prefix 
(25) 



u 


0 


2 





Number 
(26-29) 



□ 

Suffix 
(30) 



TOTAL MONTHS IN 
YOUR OUTY AFSC 



TOTAL MONTHS 
ACTIVE FEOERAL 
MILITARY SERVICE 



1 


2 


2 


(31*33) 




M 


2 


5 



(34-36) 



SEX 

(Check one) 



□ □ 

M F 
1 2 (37) 



, LEVEL OF YOUR 

JOB WITHIN 
YOUR ORGANIZATION 



(40) 
□ 7 



Command 
Element 
Directorate. 
DePt, Office 
(or Equlv) 

Division (or Equlv) £1 5 

Branch 
(or Equiv) 

Section 
(or Equlv) 

Unit , f-i 

(or Equlv) LJ 2 

Other (5pecify) 1 



□ 6 



□ 4 

□ 3 



HIGHEST LEVEL 
OF YOUR 
EOUCATION 



Non High 
School Grad 

High School 
Grad 

1 year 

College 

2 Years 
College 

3 Years 
College 

Bachelor's 
Degree 
Some Post- 
Grad Work 

Master's 
Degree 

Doctoral 
Degree 



(41) 

□ 1 

□ 2 

□ 3 

□ 4 

□ s 

□ e 

(Xl7 

□ a 

□ 9 



YOUR MAJOR FIELO 
OF STUOY 



Education 



Physical 
Arts, 

Humanities 

Engineering 

Business, 
Management 

Medical, 
Legal 

Social 
Science 

Military 
Science 

Other (Specify) 



(42) 

□ 1 

□ 2 

□ 3 

□ s 

□ e 

□ 7 

□ b 

9 



COMPLETED BY: 



WITHHELD . 

Signature 



WITHHELD 

Duty PhOne p^te 



REVIEWED BY: 



SUPER VISOR CHECK ONE -> 



WITHHELD : 

Slgnaturt of lmm«dtat« Sup«rvlsor 



.WITHHELD 

DAFSC 



MAJOR AIR 
COMMAND 



(38) 

□ a 

□ 



AAC 

USAFA I I B 

ADC Q C 

USAFG Q 0 

AFAFC Q E 

AFLC Q F 

AFSC □ H 

ARPC D I 

ATC □ J 

AU □ K 

USAFSO n L 

AFRES □ M 
Hq USAF Q N 

AFDAA □ O 
HqComd P 

MAC □ Q 

PACAF Q R 

SAC Q S 

TAC O T 

USAFSS □ U 

AFCS □ V 

Other Q 2 



Specify 



SUPERVISOR'S 
JUDGMENT 
OF MOST 
APPROPRIATE 
GRAOE FOR 
THIS JOB 
(43) 



Lt 

CaPt 

Ma] 

L/C 

Col 

Gen 



□ 2 

□ 3 



Pai9e4 



Fig, CI {Continued) 
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